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Abstract. A class of robust estimators which are obtained from 
dual representation of ^-divergences, are studied empirically for 
the normal location model. Members of this class of estimators 
are compared, and it is found that they are efficient at the true 
model and offer an attractive alternative to the maximum likeli- 
hood, in term of robustness . 
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1. Introduction 

Divergences of probability distributions are widely used in a number 
of theoretical and applied statistical inference, they are also of key 
importance in data processing problems, see Basseville (2010). The 0- 
divergence modeling has proved to be a flexible and provided a powerful 
statistical modeling framework in a variety of applied and theoretical 
contexts, see Liese and Vajda (2006, 1987), Pardo (2006), Broniatowski 
and Keziou (2009) and the recent monograph by Basu et al. (2011). 
Recently introduced, the minimum divergences estimation method based 
on a dual representation of the divergence between probability mea- 
sures, is an appealing estimation method, it avoids the use of non- 
parametric density estimation and the complications related to the 
bandwidth selection. The estimators are defined in an unified way for 
both continuous and discrete models. They do not require any prior 
smoothing and include the classical maximum likelihood estimators as 
a benchmark. These estimators called "dual 0-divergence estimators" 
(D0DE's), was shown by Keziou (2003) and Broniatowski and Keziou 
(2009), under suitable conditions, to be consistent, asymptotically nor- 
mal and asymptotically full efficient at the true model. 
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Application of dual representation of 0-divergences have been con- 
sidered by many authors, we cite among others, Keziou and Leoni- 
Aubin (2008) for semi-parametric two-sample density ratio models, ro- 
bust tests based on saddlepoint approximations in Toma and Leoni- 
Aubin (2010), bootstrapped 0-divergences estimates are considered in 
Bouzebda and Cherfi (2011), extension of dual 0-divergences estimators 
to right censored data are introduced in Cherfi (2011), for estimation 
and tests in copula models we refer to Bouzebda and Keziou (2010) 
and the references therein. 

The 0-divergences estimators are motivated by the fact that a suitable 
choice of the divergence may lead to an estimate more robust than the 
MLE one, see Jimenez and Shao (2001). Toma and Broniatowski (2010) 
studied the robustness of the D0DE's through the influence function 
approach. However the fact that the D0DE's have unbounded influence 
functions in the normal location model, causes some concern that these 
estimators are non-robust. In this article we show that the e- influence 
functions are more suitable to capture the robustness properties of the 
D0DE's. We also give some insights about the choice of the escort 
parameter. Simulation results indicate that the D0DE's attain the 
dual goal of robustness and efficiency, they are competitive with Huber 
M-estimator without the loss of efficiency at the true model. 
The rest of the paper is organized as follows. In Section 2 we pro- 
vide background material concerning the dual ^-divergences estimators. 
Section 3 is devoted to the choice of the escort parameter. Section 4 
deals with robustness. In section 5, we illustrate the performance of the 
method in real data example. A simulation study described in Section 
6 investigates the asymptotic properties of the estimators. Section 7 
contains some concluding remarks. 



The class of dual divergences estimators has been recently introduced 
by Keziou (2003), Broniatowski and Keziou (2009). In the following, 
we shortly recall their context and definition. 

Recall that the 0-divergence between a bounded signed measure Q and 
a probability P on @, when Q is absolutely continuous with respect to 
P, is defined by 



where is a convex function from ] — oo, oof to [0, oo] with 0(1) = 0. 

Well-known examples of divergences are the Kullback-Leibler associ- 
ated to the function <fi(x) = x logx— x+1, modified Kullback-Leibler for 



2. Background and estimators definition 
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4>{x) = — logx + x — 1, the x 2 -divergence with <f>(x) = \{x — l) 2 and the 
Hellinger distance given by 4>(x) = 2(^/x — l) 2 . All these divergences 
belong to the class of the so called "power divergences" introduced in 
Cressie and Read (1984) (see also Liese and Vajda (1987) chapter 2). 
They are defined through the class of convex functions 

x^ — *yx ~\~ *y — 1 

x G]0, +ooh> <f> 7 {x) : = / -{ (1) 

717 - 1) 

if 7 G R \ {0, 1}, 4>o(x) := — logx + x — 1 and <f>i(x) := x logx — x + 1. 
(For all 7 e 1, we define 7 (O) := lim^o 4 > 'y{ x ))- So, the KL- divergence 
is associated to <pi, the KL m to <po, the \ 2 t° 02 and the Hellinger 
distance to <pi/2- We refer to Liese and Vajda (1987) for an overview 
on the origin of the concept of divergences in statistics. The reader 
interested in other class of (^-divergences can refer to the recent paper 
of Kus et al. (2008), which propose a simple method of construction of 
new families of 0-divergences. 

Let Xi, . . . ,X n be an i.i.d. sample with p.m. Pg . Consider the prob- 
lem of estimating the population parameters of interest 9o, when the 
underlying identifiable model is given by V = {Pe : 6 G 0} with 
a subset of M. d . We denote A a dominating measure for V and the 
resulting density of any Pq is denoted pg. 
Let 4> be a function of class C 2 , strictly convex and satisfies 



p a (x) 



dP (x) < oo. (2) 



By Lemma 3.2 in Broniatowski and Keziou (2006), if the function cf) 
satisfies: There exists < r\ < 1 such that for all c in [1 — r), 1 + 77], we 
can find numbers Ci, C2, C3 such that 

4>{cx) < ci<f>(x) + C2 \x\ + C3, for all real x, (3) 

then the assumption (2) is satisfied whenever D^Pg, P a ) is finite. From 
now on, we suppose that there exists a neighborhood U of 9q for which 
D^Pg, P a ) < 00 whatever 9 and a in U. Note that all the real convex 
functions 7 pertaining to the class of power divergences defined in (1) 
satisfy the condition (3). 
Under the above conditions, the 0-divergence: 



D (i> (Pg,P eo ) = J <t>[^\ dPe 



) • 



can be represented as the following form: 

D^(Pg, Pg Q ) = SUp / h(9, a) dPg Q , (4) 

OL&A J 
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where h(9, a) : x i-> /i(6*, a, x) and 

p (x) ,, / p e [x) \ , I p e {x) 



h(9,a,x):= I 4>[y) dP e 



Pa{X) \Pa{X) J \p a {X 



(5) 

Since the supremum in (4) is unique and is attained in a = 9q, in- 
dependently upon the value of 9, define the class of estimators of 9q 
by 

a$(9) := argsup / h(9,a)dP n , 9 e 6, (6) 

a&J J 

where h(9,a) is the function defined in (5). This class is called "dual 
(^-divergences estimators" (D^DE's). 

The corresponding estimating equation for the unknown parameter is 
then given by 



J ^h(6,a) dP n = 0. 



(7) 



Remark that the maximum likelihood estimate belongs to the class 
of estimates (6). Indeed, it is obtained when <p(x) = — \ogx + x — 1, 
that is as the dual modified i^L-divergence estimate. Observe that 

J h(9, a)dP n - - J log (JM dP n . 

Hence keeping in mind definitions (6), we get 

®KL m {9) = argsup - / log ( — ] dP n = argsup / log(p Q )dP n = MLE, 
oee J \PaJ aeeJ 

independently upon 9 

Formula (4) defines a family of M-estimators indexed by some instru- 
mental value of the parameter 9 and by the function <ft defining the 
divergence. In the sequel we call 9 the "escort parameter", the choice 
of 9 appears as a major feature in the estimation procedure, see Section 
3 below. 

Recall that an M-estimator of ^-type is the solution of the vector 
equation: 

JiP(x;a)dP n = 0, (8) 

where the elements of ip{x] a) represent the partial derivatives of h(9, a, x) 
with respect to the components of a. For more details about M- 
estimators we may refer to Huber (1981), Hampel et al. (1986), Maronna 
et al. (2006) and the references therein. 
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We apply the dual representation of 0- divergences (4), which we spe- 
cialize to the present setting. Consider now, the normal model with 
density: 

PeA x ) = —7^ e *v\-\\- — -J }, (9) 



aV2n ! 2 \ a 
and the power divergences family (1). Observe that, for 7 e K\ {0, 1} 

1 ffPeA^Y' 1 ,s A 1 a^a-^ ( 7 ( 7 -l)(0- a ) 2 

p e>(T (x)dx= ==exp 



1 -lJ \p a ,s{x) J ' 7-1 v^ 2 - (7 - l)^ 2 1 2(7^ - (7 - I)* 2 ) 

Hence, 

^ ,„ ^ [ 1 S^o-M f 7 ( 7 -l)u9-a) 2 

L> 7 (P,9 ff , P n ) = sup < —j== , 1 = exp <^ ? — 

a,5 [7-1 - (7 - l)a 2 1 2( 7 <r 2 - (7 - l)^ 2 ) 

w exp l 2 \v o- / v ? ; ;j 7(7 

For 7 = 0, 

X-0\ 2 /X-a\ 2 \ , fa 



^<W0 = ^E((^) -(^) )-io Su 

For 7 = 1, 



1/ f <j\ 2 f 9 — a\ 2 \ , / cr 



1 - 

i E 



a\ | 1 / fX 4 -9\ 2 /X- -a x 2 
exp 



n A — ' \o l \ \ a J \ a 

i=l 



For the normal family Pg = Af(9, 1), with the location parameter 9 and 
scale (7 = 1. It follows that, for 7 £ R \ {0, 1}, 

D 7 (P e ,P n ) : = sup I h(9,a)dP n 



1 |- 7 ( 7 _i)(e_ a ) 2 
sup < exp 



7-1 
1 n 

- 5>xp - a)(0 + a - 2X,)} 



7 n ^ I 2 ' 1 ) 7(7- 1) [ ' 
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|^i>-") (e + a-2X i )\ 



For 7 = 0, 

D KLm (P e ,P n ) := sup / h (9, a) dP n 

a 

= sup 

a 

For 7 = 1, 

D KL {Pg,P n ) := sup [h(9,a)dP n 

a J 

f 1 1 - f 1 

= sup < -- (0 - a) 2 J^exp <^ -- (0 - a) (9 + a - 2X t 

a [ 2 n . =l { 2 

We remark that the above optimizations are a feasible computationally 
closed-form expressions and can be performed by any standard non 
linear optimization code. 

3. On the choice of the escort parameter 

The very peculiar choice of the escort parameter defined through 9 = 9 
has same limit properties as the MLE. The D0DE (9q), in this case, 
has variance which indeed coincides with the MLE, see for instance 
(Keziou, 2003, Theorem 2.2, (1) (b)). This result is of some relevance, 
since it leaves open the choice of the divergence, while keeping good as- 
ymptotic properties. For data generated from the distribution Af(0, 1), 
Figure 1 shows that the global maximum of the empirical criterion 

P n h \ 9 n , a j is zero, independently of the value of the escort parameter 

9 n (the sample mean X in Figure 1(a) and the median in Figure 1(b)) 
for all the considered divergences. 

Unlike the case of data without contamination, the choice of the escort 
parameter is crucial in the estimation method in the presence of out- 
liers. We plot in Figure 2 the empirical criterion P n h U) n ,ai\, where 
the data are generated from the distribution 

(1 - e)A/"(#o, 1) + e5 10 , 
where e = 0.1 and 9 = 0. Figure 2(a) illustrates the empirical criterion 
under contamination, when we take the empirical mean, 9 n = X, as the 
value of the escort parameter 9, it shows how the global maximum of the 

empirical criterion P n h [9 n , c^j shifts from zero to the contamination 

point. In Figure 2(b), the choice of the median as escort parameter 
value leads to the position of the global maximum remains close to 
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(a) ^ (b) 

3„- -0.056541 17 0„- -0.1898001 




-2-1012 -2-101 



Figure 1. Criterion for the normal location model. 



(a) (b) 



0„- 0.9251538 8„ - 0.163952 




-2-1012 -2-101 



Figure 2. Criterion for the normal location model un- 
der contamination. 
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a = for Hellinger (7 = 0.5), x 2 (7 = 2) and fCL-divergence (7 = 
1), while the criterion associated to the KL m -divergence (7 = 0, the 
maximum is the MLE) stills affected by the presence of outliers. 
In practice, the consequence is that the escort parameter should be 
chosen as a robust estimator of #0, say 6 n . 

Observe that for the power divergences family (1), the estimating equa- 
tion (7) reduces to 

-/(^) , " , i1*) t+1 i:(ifi) , 'l= 1 ' < io > 

J \Pa{x)J p a {x) n^\p a {Xi)J p a {Xi) 

and the estimate a^(#) is the solution in a of (10). An improvement 
of the present estimate results in the plugging of a preliminary robust 
estimate of 8 , say 8 n , as an adaptive escort parameter 9 choice. 




Pe ( x ) 

Figure 3. Behaviour of the ratio — 2— — under con- 

Pe {x) 

tamination, for a randomly generated Normal sample 
Af(6o, 1) °f srze 100 with 10% of contamination by 10. 

Let x be some outlier, the role of the outlier x in (10) appears in the 
term 



x 



7 



Pa \ X ) (11) 
Pa{X) J Pa{x)' 

The estimate 0.^(6) is robust if this term is stable. That is, if it is small 
when a is near 9 . If the escort parameter 9 n is not a robust estimator, 
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the ratio — 7} — — can be very large, see Figure 3. This is due to the fact 
Pe {x) 

that the outlier x will be more likely under Pg , that is 8 n will lead to 
an over evaluation of pg- (x) with respect to the expected value under 
#o, say pe (x). To guard against such situations, compensate through 
the choice of 7, this requires further investigations. 



4. Robustness 

One method of assessing the robustness of an estimator is by consid- 
ering its influence function (IF), with the usual interpretation being 
that a robust estimator will have a bounded influence function. How- 
ever this requirement makes the corresponding estimator deficient at 
the model in relation to the maximum likelihood estimator which have 
unbounded influence function for most common models. 
As we will see in the following, our class of dual {^-divergences esti- 
mators have strong robustness features in spite of having the same 
influence function as the maximum likelihood estimator. On the other 
hand, being equal to the influence function of the MLE, the influence 
function of the D0DE's is potentially unbounded. Thus the robustness 
of the D0DE's cannot be described through the traditional bounded 
influence approach. Hampel (1974) claims that the use of the e-IF 
(before the limit) is preferable to the use of the influence function to 
assess estimator robustness. The limiting form is often used because it 
is usually easier to evaluate, and it does not depend on e. Beran (1977) 
claims that for evaluating the robustness of a functional with respect 
to a gross-error model, one should consider the e-influence function in- 
stead of the influence function, unless the former converges to the latter 
uniformly. For more details, see also Section 4.7 of Basu et al. (2011). 
In this section we present the e-influence function technique and show 
that there is no intrinsic conflict between the robustness of our estima- 
tors and optimal model efficiency. 

The statistical functional associated with the estimate a^(9) is given 
by 

T e (P Bo ) = argsup / h(8,a) dP 0o , (12) 
aee J 

which is Fisher consistent, namely Tg{P a ) = a for all a e O, keeping 
in mind (6). 

The influence function of the functional Tq is defined as 

IF(x; T, Fi.) = lim MLzil^±i^j™, 
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in which 8 X is the Dirac measure at point x. The e-influence function 
is the quotient 

Trp/ \ T e ((l-e)P do + e5 x )-9 

e — Lb (x) = . 

e 

Using existing theory on M-estimators, see Hampel et al. (1986), p. 
230, see also Proposition 1 in Toma and Broniatowski (2010), the in- 
fluence function of the functional Tg corresponding to an estimator 
a $(6) is given by 

lF(x;T e ,Pe ) := -ST l \ U" (^-) -^PfcdP, 

[J \PeJ Pe 

(13) 



When 9 — 9q, it reduces to the influence function of the MLE given by 

lF(x;T eo ,P 6o ) = I eo lP ^ y 
where Ig is the information matrix defined by 

J Pe 

Using (13), the influence function for the normal location model is, for 
7 GM\{0,1}, 

IF(x; T e ,P 6o ) : = 

For 7 = (KL m ), 

lF(x;T e ,Pg ) := x - 9 . 

For 7 = 1 (KL), 

m , r p , (x - 9 ) exp {-1(9 - 9 )(9 + 9 - 2x)} + (9 - 9) 
lF(x;T 6 ,P eo ) := 1 + (g - e f ' 

Remark that when 9 = 9o, the influence functions of Tg coincide for all 
7 and are equal to the influence function of the MLE, 

IF(x;MLE,P eo ) := x - 9 . 

Figure 4, presents the influence functions of the dual 0-divergences 
estimators in the normal location model, with 9 = 0.1 and 9 = 0.5. We 
can see that the influence functions are unbounded. Thus, the D0DE's 
are examples of estimators for which the limiting form of their influence 



{x - 9 ) exp {-1(9 - 9 )(9 + 9 - 2x)} + 7(^0 - 0) exp \ 




(l + 7 2(0_0 o )2) exp j 


f 7(7-l)(0-0 o ) 2 l 

L 2 J 


\ 
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Figure 4. Influence functions of the D0DE's for the 
normal location model. 



functions does not reliably provide information about the form of the 
e-IF's. 

Empirical e-IF's were calculated for e's= 0.1, 0.2. This was done by 
generating a sample size of 100 for Normal distribution J\f(9o, 1) with 
#o = l- After the data set was sorted, the largest en values were then 
iteratively reassigned along a grid of values ranging from 0.01 to 25. At 
each grid value the D^DE's were computed. This was repeated 1000 
times. The mean of the D0DE's at each grid were plotted against the 
grid. Below are the plots of the empirical average value of the D^DE's, 
see Figures 5 and 6. 

Figure 5(a) illustrates the empirical e-IF, when we take the mean, 
9 n = X, as the value of the escort parameter 9, it shows that the 
D0DE's a$(9) have the same empirical e-IF as the MLE. In Figure 
5(b), the choice of the median as escort parameter value improve con- 
siderably the robustness properties of our estimators. The D0DE's, 
with robust escort parameter perform very well, outliers tend have less 
influence on the D0DE's. Clearly, we can see from Figure 6 that the 
MLE is greatly affected by the value of the outlier. 
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Figure 5. Empirical e-influence functions, e = 0.1. 



5. A Real data example 

This example involves Newcombs light speed data (Stigler (1977), Ta- 
ble 5). The data were also analysed by Brown and Hwang (1993), who 
were trying to fit the best approximating normal distribution to the 
corresponding histogram. The data set shows a nice unimodal struc- 
ture, and the normal model would have provided an excellent fit to the 
data except for the two large outliers. 

For the dataset, Table 1 gives the values of the D0DE's. These estima- 
tors exhibit strong outlier resistance properties. The histogram, and 
the normal fits using the maximum likelihood estimate and the D0DE's 
are presented in Figure 7, all the normal densities fit the main body of 
the histogram quite well, except the MLE. Note that the values of the 
escort parameters considered in this example are the median for 9 and 
the mad for a. 



6. Simulation 



In this section, we present results of a simulation study which was 
conducted to explore the properties of the D0DE's. 
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X 



Figure 6. Empirical e-inffuence functions, e = 0.2. 



7 

0.5 1 2 
a 26.21 27.67 27.00 27.64 

a 10.66 5.16 4.47 4.84 
Table 1. Estimated parameters for the Newcomb data 
under the normal model. 



A sample is generated from N{6q, 1) with 8 = 0. The D0DE's are 
calculated for samples of sizes 25, 50, 75, 100, 200 and the hole pro- 
cedure is repeated 1000 times. In the examination of the robustness of 
the D0DE's, the data are generated from the distribution 

(l-e)JV(0o,l) + e*io, 

where e = 0.05, 0.1, 0.2. The value of escort parameter 9 is taken to 
be the median. 
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A histogram of the Newcomb data 




Figure 7. A histogram of the Newcomb data with nor- 
mal fits. 

The values of 7 are chosen to be 0, 0.5, 1, 2 which correspond to 
the well known standard divergences: K L m — divergence, the Hellinger 
distance, KL and the \ 2 ~ divergence respectively. These estimators are 
also compared with some other methods, including maximum likelihood 
estimator (MLE) and Huber estimator (see Huber (1964)) using the ip- 
function 

W ~ \ rsgn(t) if \t\ > t 

for some constant r > 0. A value of 1.4 was used for r, for location 
estimation which is in the range of values shown to perform well in the 
Princeton Robustness Study (see Andrews et al. (1972)). We carried 
out this analysis within the R Language R Development Core Team 
(2009). 

Under the true model, from Table 2, as expected, the MLE produces 
most efficient estimators in this case. The D0DE's seem to be a good 
competitors to the MLE in terms of MSE. Recall that, theoretically, the 
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Table 2. MSE of the estimates under the true model. 



n 



25 50 75 100 150 200 



7 



0.0386 


0.0187 


0.0129 


0.0096 


0.0064 


0.0055 


0.5 


0.0386 


0.0187 


0.0129 


0.0096 


0.0064 


0.0055 


1 


0.0386 


0.0187 


0.0129 


0.0096 


0.0064 


0.0055 


2 


0.0440 


0.0190 


0.0129 


0.0096 


0.0064 


0.0055 


M-Estimator 


0.0400 


0.0199 


0.0136 


0.0100 


0.0067 


0.0057 



Table 3. MSE of the estimates under 10% of contamination 



n 



25 50 75 100 150 200 



7 



1.3999 


1.1696 


1.1041 


1.0941 


1.0505 


1.0360 


0.5 


0.2004 


0.1610 


0.1509 


0.1475 


0.1447 


0.1402 


1 


0.1280 


0.0874 


0.0780 


0.0726 


0.0689 


0.0649 


2 


0.1393 


0.0843 


0.0731 


0.0680 


0.0638 


0.0598 


M-Estimator 


0.2460 


0.0902 


0.0724 


0.0646 


0.0593 


0.0540 



Table 4. MSE of the estimates under 20% of contamination 



n 



25 50 75 100 150 200 



7 



4.6936 


4.3133 


4.1415 


4.1344 


4.1282 


4.0724 


0.5 


0.4935 


0.4351 


0.3947 


0.3802 


0.3812 


0.3701 


1 


0.3253 


0.2673 


0.2325 


0.2178 


0.2163 


0.2067 


2 


0.3092 


0.2433 


0.2072 


0.1929 


0.1906 


0.1812 


M-Estimator 


2.6171 


1.5906 


1.1710 


0.9353 


0.7837 


0.6220 



D^DE's are asymptotically efficient. It is interesting to note that the 
asymptotics appear to take effect for a sample sizes of 75 or more but 
the results for sample sizes below 75 are also good, thus the D0DE's 
perform as well as the MLE at the true model. 

We now turn to the comparison of these various estimators under con- 
tamination. We can see from Table 3 that for small sample sizes the 
D0DE's yield clearly the most robust estimates . As n increases, the 
M-Estimator obtain slightly higher performance. For high amount of 
contamination, we can see from Tables 4 and 5 that the D0DE with 
7 = 2 has the smallest MSE over all other D0DE's and outperform the 
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Table 5. MSE of the estimates under 25% of contamination 



25 50 75 100 150 200 



7 



7.1318 


6.6138 


6.5256 


6.3594 


6.3376 


6.3436 


0.5 


1.1670 


0.6124 


0.5920 


0.5563 


0.5596 


0.5533 


1 


0.9946 


0.3963 


0.3754 


0.3442 


0.3454 


0.3383 


2 


0.9921 


0.3561 


0.3347 


0.3047 


0.3051 


0.2980 


M-Estimator 


5.7497 


4.3835 


3.9891 


3.7579 


3.5211 


3.3328 



MLE substantially and the performance of the M-Estimator decreases 
dramatically. 

7. Conclusions 

The aim of this paper was to investigate the new estimation procedure 
based on the dual representation of 0-divergences for the univariate 
normal location model. The estimators are easily computed and exhibit 
appropriate asymptotic behavior. First, we evaluated the impact of the 
choice of the escort parameter on the estimates and provided a practical 
choice of the escort parameter. Second, we have shown that there is no 
intrinsic conflict between the robustness of our estimators and optimal 
model efficiency, by considering empirical e-influence functions. 
The simulation results presented here provide solid evidence that the 
D0DE's in the normal location setting, obtain full efficiency at the 
true model while keeping good performances as robust estimators, 
they demonstrated unexpected empirical robustness for high amounts 
of contamination. Thus they provide good alternative to maximum 
likelihood. 

In this paper we have limited ourself to the D0DE's associated to the 
standard divergences, which are widely used in statistical inference. It 
will be interesting to investigate theoretically the problem of the choice 
of the divergence which leads to an "optimal" estimate in terms of 
efficiency and robustness, we leave this study open for future research. 
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